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Abstract — Convolutional Low-Density-Parity-Check (LDPC) 
ensembles have excellent performance. Their iterative threshold 
increases with their average degree, or with the size of the 
coupling window in randomized constructions. In the later 
case, as the window size grows, the Belief Propagation (BP) 
threshold attains the maximum-a-posteriori (MAP) threshold of 
the underlying ensemble. In this contribution we show that a 
similar phenomenon happens for the growth rate of coupled 
ensembles. Loosely speaking, we observe that as the coupling 
strength grows, the growth rate of the coupled ensemble comes 
close to the concave hull of the underlying ensemble's growth 
rate. For ensembles randomly coupled across a window the 
growth rate actually tends to the concave hull of the underlying 
one as the window size increases. Our observations are supported 
by the calculations of the combinatorial growth rate, and that of 
the growth rate derived from the replica method. The observed 
concavity is a general feature of coupled mean field graphical 
models and is already present at the level of coupled Curie- 
Weiss models. There, the canonical free energy of the coupled 
system tends to the concave hull of the underlying one. As we 
explain, the behavior of the growth rate of coupled ensembles is 
exactly analogous. 

I. Introduction 

Convolutional low-density parity-check (LDPC) codes 
where initially introduced by Felstroem and Zigangirov [Q] 
and since then they have spurred a large body of work (see 
Ha, 0, S, mi, ig, m, 0, mi, Uni). The constructions 
of convolutional LDPC ensembles involve the appropriate 
coupling, of standard individual LDPC ensembles, into a one 
dimensional chain. The main observation is that as the strength 
of the coupling (e.g. the average degrees or the size of the 
window) increases, the BP threshold of the coupled ensemble 
comes close to the MAP one of the underlying individual 
ensemble. This provides a new versatile way of approaching 
capacity. 

In this contribution we focus on the growth rate of the 
relative Hamming distance spectrum for coupled ensembles of 
LDPC codes (we will call this simply the growth rate in what 
follows, see sec. III-AI for definitions). We study this quantity 
for two kinds of ensembles, by combinatorial as well as replica 
methods. Both of these approaches have been already used for 
standard LDPC ensembles (see O], Qa), lEl, IHI, JTtI ). 
The growth rate of a generic standard LDPC ensemble is not 
a concave curve and displays a gap near weight, indicating 



the absence of code words of macroscopic weight near 0. 
Here we show that, as the coupling strength increases, the 
growth rate of the convolutional LDPC ensembles becomes 
nearly identical to the concave hull of the growth rate of the 
individual ensemble. This phenomenon is illustrated in fig. 
[T] We conjecture that, for the ensembles with a randomized 
coupling accross a window of size w, in the limit where 
the length of the chain L — > +00 (first) and w +00 
(second) the growth rate really tends to the concave hull of 
the individual system curve. 

The threshold saturation phenomenon is a general feature 
of coupled chains of mean field graphical models. The same 
phenomenon happens in Curie- Weiss (CW) chains lT6l . and 
also coupled constraint satisfaction models ifTSl . As explained 
in more detail in sec. III-Dl the concavity of the growth rate for 
coupled LDPC chains is the precise analog of the concavity of 
the canonical free energy of a coupled CW chain. This analogy 
in fact constituted our initial intuition for the behavior of the 
growth rate of coupled LDPC models. 

II. Models and Results 
A. Notations and Preliminaries 

For the underlying individual ensemble we take a regular 
LDPC(r7,, /, r) ensemble with m = check nodes of degree 
r and n variable nodes of degree I. We define the relative 
weight of a code-word (xi, x„) as w = 1 — ^ Tl!i=i ^'^'^ 
call An{u!) the number of code words of relative weight lu. 
Note that here u £ [— 1, • ■ • ,1] with uj = 1 corresponding to 
the all codeword. The growth rate is 



G„(w) = iE[lnA„(w)] 



(1) 



where the expectation is over the LDPC(n, Z,r) ensemble. 

Applying Jensen's inequality one obtains an upper bound 

G„(a;) < Gn,coni{oj), sometimes called the combinatorial 
growth rate. 
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G„,com(w) = -lnE[A„(a;)] 



(2) 



Here we are interested in the infinite length quantities G{uj) 
and G'com('^)- For the regular ensemble considered here the 



infinite length limit^ are believed to be equal: indeed the 
combinatorial calculation of G'com('^) and the replica method 
calculation of G{uj) lead to the same set of equations (see ifTTl 
and for a partial proof lITSi ). One first fixes lu and solves for 
{y,z,h) the set of equations 



tanh(/i tanh"^(y)), 
tanh{h + Ztanh""'^ y). 



(3) 



For uj = 1 there is the family of trivial solutions {y = z = 
l,h G M); for ui < I there exist a non-trivial single valued 
solution (yiio), z{u!), h{u!)) from which one can compute 



i In(i^) + ln(e''(l + yY + e-\l - yY) 
nn{l + zy) - ujh (4) 



Both G{uj) and h{uj) are illustrated as dotted lines on fig. |3] 
for a (3, 6) ensemble. In the first three terms one finds the 
contributions of checks, n variable nodes, and nl edges. 
It is easily checked that the total derivative satisfies 



dGjio) 
duj 



-h{uj). 



(5) 



This formula is useful because it allows to reconstruct the 
growth rate by integrating h{oj) (note G(l) = 0) 



G(w) 



dLL)h{uj). 



(6) 



B. The coupled ensembles 

The {l,r,L) ensemble. We assume that j is integer At 
each position i G {—L, • • ■ ,i} of a one-dimensional chain 
(of length 2L + 1) we lay down sets of m checks and n 
variable nodes. At the right boundary, we also add additional 
sets of m check nodes in positions L + 1, - ■ ■ ,L + / — 1. Each 
check (resp. variable) in the range [— L, • • • ,L] has a set of r 
edges that we view as a I groups of j edges each. Fix i. For 
each k E {0, ■ ■ ■ ,1 ~ 1}, n edges emanating from n variable 
nodes at position i are connected - through a uniformly random 
permutation - to a group of TOj edges emanating the checks at 
position i + k. The couple {i, i + k) will parametrize the class 
of edges connecting variable nodes at position i to checks at 
position i + k. 

The {l,r,w,L) ensemble. This ensemble is a modification 
of the previous one obtained by adding a randomization of the 
edge connections. We refer to ||9] for the detailled construction. 
It is no longer assumed that j is an integer, and there is 
a new parameter w G N, the window size. In the limit 
of large size, each edge of a variable node at position i is 
uniformly and independently connected to a check in the 
range {i — w + I,-- - and each check at position i is 
uniformly and independently connected to a variable node in 
{?,•■• ,i — w + l}.lt can be shown that for n and m large the 
connections of a variable nodes at position i towards checks 

'it is an open problem to prove the existence of the first limit. 



are independ and uniform in the range {i, - ■ ■ ,i + w — 1} and 
that variable nodes have constant degree I. 



The relative weight ui of code words (xj,j)^i'i=-L 



1 ^ 
2L+ 1 /-^ 



uj{z) = 1 



(7) 



and the growth is defined as in ([U but with the normalization 
nL in front of the log. We denoted it by Gi(aj) for the 
{l,r,L) ensemble and G{uj2) for the {l,r,w,L) ensemble. The 
corresponding combinatorial growth rates are Gi,com('^) and 
G2, 00111(1^). 

C. Main result: concavity of the growth rate for the coupled 
ensembles 

The {l,r,L) ensemble. As for a usual {l,r) ensemble IfTTl , 
in turns out that for the {I, r, L) the replica and combinatorial 
approaches yield the same expressions (up to boundary terms 
that vanish as i — > +00) for the growth rate. In sec. |IIT] we 
sketch the combinatorial derivation which leads to a set of 
fixed point equations, which we have solved numericaly. Fig- 
ure [T] shows that the growth rate for the (3, 6, 256) ensemble 
is indistinguishable from the concave hull of the underlying 
ensemble. Note that the picture is symmetric for lo negative 
because the check node degrees are even. Figure |3] shows 
both curves Gi(a;) and hi{uj) for the (3,6,32) ensemble. 
It is clear that the concave hull of Gi(a;) develops hand in 
hand with the plateau in hi{uj). The plateau is very close to 
a value he = h{ijjc) where lOc is found from the concave hull 
construction for G(a;) (i.e it is the tangency point of a straight 
fine going through w = 1). Note that it can also be obtained 
directly from the h{ijj) dotted curve by a Maxwell equal area 
construction. A magnification of the curves shows that they 
display wigles of small amplitude. There are approximately 
2L oscillations with a period 0(-^) that extend throughout 
the plateau of hi{ijj) and throughout the straight slope in 
Gi(a;). The amplitude of the wiggles in Gi(cj) is -j- times 
the amplitude of the wigles for hi{uj) and therefore harder to 
detect. Fig. [T| shows the sequence of curves as a function of 
L. The limiting curve is numericaly very close to the concave 
hull of the underlying standard (3, 6) ensemble growth rate. 
Note however that there is no fundamental reason for it to be 
concave and this is not exactly the case. As explained below 
we conjecture that concavity is attained only in an appropriate 
limit with the {I, r, w, L) ensemble. 

The {I, r, w, L) ensemble. In the randomized construction 
the combinatorial and replica expression are not the same, 
although they are numericaly close. It is generaly believed 
that the replica expressions are exact and we therefore stick 
those. In sec. |IV] we give the result of the replica calculation. 
The corresponding fixed point equations involve non-trivial 
densities and we have solved them by the method of population 
dynamics. The curves h2{uj) and G2(w) will display the same 
features as in the previous ensemble so do not comment more 
on these. Our interest is in the behavior of the model as w 
grows (when L is formaly infinite). For h > h^^'^^ there is 
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a unique trivial fixed point whereas for h < h[^'^'^ two extra 
fixed points (besides the trivial one) appear. Table U gives the 
values of the bifurcation point for w = 1 (uncoupled case), 
w ~ 2 and w = 3 when L = 20. As w grows /il^'^' he 
and we conjecture that lim„,„>+oo limL-i.+oo h\^' — he- As 
a consequence, for the [l, r, w, L) ensemble we conjecture that 

1 



lim lim lim — E[lnA*j^^ 



(^)] 



concave hull G{lo) 
(8) 

We stress that the order of limits matters. Note that here we 
expect from the experience with the coupled CW chain (sec. 
III-DI l that the amplitude of the wiggles is exponentialy small 
in w but this is difficult to observe. 
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Fig. 1. Left: Evolution of Gi(aj), uj G [0, 1], for (3, 6, L) ensembles 
with L = 4, 8, 16, 32, 64, Right: (3, 6, 256) ensemble 
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Fig. 2. Giiuj) for u) G [0,1]. Dotted curves: underlying (3,6) 
ensemble. Full curves: (3, 6, 32) ensemble 



D. Analogies with the CW chain 

We briefly summarize the picture that arises in the simple 
setting of the coupled chain of Curie- Weiss models IfTSl . 
|fT6| . The Curie- Weiss model (CW) in the canonical ensem- 
ble (or lattice-gas interpretation) has Hamiltonian Hj^ = 
^W^it u) ^tSu where the spins st = ±1 are attached to the 

vertices of a complete graph, the sum over {t, u) carries 
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Fig. 3. hliu)) for for u G [0, 1]. Dotted curves: underlying (3, 
ensemble. Full curves: (3, 6, 32) ensemble 
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TABLE I 

Thresholds of the underlying (r, /) ensemble; and 

ITERATIVE THRESHOLDS OF (r, I, w, 20) ENSEMBLES 



over all edges of the graph and J > 0. The free energy, for a 
fixed magnetization m = -i- X^tli ^t' is easily computed 



lim — In 

St'.m fixed 



N 



$(to) = — — — h2{m) 
(9) 

where /i2(m) is the binary entropy BemoulU(^^^). The van- 
der-Waals curve (or isotherm) is 



h{m) 



(i<f>(m) 
dm 



-Jm 



2 1 



(10) 



This relation can also be expressed as a fixed point equation 
for m as a function of h, namely m = tanh(jTO + h), 
the CW equation. Fig. |4] shows $(771) and h{m) for J > 1 
(dotted lines). The van-der-Waals curve (fTOl i depicted on figure 
|4] has two important thresholds: the static phase transition 
threshold at /ij, = which satisfies the Maxwell equal area 
construction and the spinodal point (or dynamic thresholds) 
which correspond to the points where the solutions of the fixed 
point equation bifurcate. 

One can consider a systems of CW models that are coupled 
along a chain: one constructs Hamiltonians where all spins 
at position i are interacting with all spins at positions [z — 
■w,z + w]. One can show that in the limit of an infinitely long 
chain, when the window size grows the canonical free energy 
of the coupled chain becomes the convex hull of $(m). This 
phenomenon and the corresponding oscillations of the van- 
der-Waals curve around the Maxwell plateau are illustrated in 
figure |4] 



LDPC ensembles can be interpreted a spin models on 
random sparse graphs with interactions between the spins 
given by hard constraints. The code words are the allowed spin 
configurations. Their relative weight w is a total magnetization 
m. The number of code words Anicd) of relative weight cj is 
a canonical partition functioiJl computed at fixed magnetiza- 
tion; the growth rate is (minus) a canonical free energy; its 
derivative h(oj) is a van-der-Waals curve. The latter has two 
thresholds: a static phase transition one given by the Maxwell 
construction at he and an iterative one h^t (the local max and 
min on fig. |4|i given by the value of h where the solutions of 
the first two fixed point equations in ^ bifurcate. The growth 
rate of coupled LDPC ensembles behaves in the same way 
than the free energy of the CW chain. 



Fig. 4. Dotted curves: free energy and van der Waals curve of the 
CW model for J > 1 as functions of m. Continuous curves: free 
energy and van-der-Waals isotherm of the coupled chain. Oscillations 
extend throughout the plateaus, have period 0{1/L) and amplitude 
exponentialy small in w. 



III. Analysis of the (r, /, L) ensemble 

For this ensemble the combinatorial and replica calculations 
yield the same set of equations. Here we sketch the main steps 
of the combinatorial method, following fT3\. We recall that 
+ k) parametrizes edges connecting variable nodes at i 
with check nodes at i + fc. We set cj' = and count the 
number of codewords of total weight W = [{2L + l)nuj'\. 
In the following expression the generating function variables 



j attached to edges (i — k, i) are set to for i — k < —L 



and i — k > 



-L. We have 

/ L l-l 



E[^„H]=^ n n 

L / l-l 

coeff n i+a^n^^Ufc 



E\ 



fc=0 



L+l-l 

coeff 



'nL=o(i 




n n 



To compute this expression in the large size limit one uses 

(n — > +CXD 

lim -coeff r/(a;)",a;*"] = inf [ln(/(a;)) - t Inx] 

n— ^+oo n ^ X 

and performs a saddle point calculation of the sum. This yields 
Gi,co„.(a.) - ^lin^ s^^P r ( E E h2{a..+^)\ 

^ In h + x n J - (2i + 1)^' In x 

i=-L \ fe=0 / 

L l-l 

^ E E^'-'+'^-^'^yvi+fe 



i — k.i / 



l-l 

n 

fc=0 



let us point out that the weight enumerator polynomial is the grand- 
canonical partition function computed for a fixed magnetic field 



inf 



i=-L k=0 



L+l-l 



E 




nL=o(i 



-i-k,!) 



'^i-k,il 



l-l 

E 

k=Q 



ai-kA 



\nz' 



i-kA 



The critical point equations are best written in terms of the 
new variables h= — ilna;, a[ ^ = 1 — 2(3;; j-, z^j- = j+F^' 

l — y' ' 

■jjij = After a lengthy calculation the critical point 

equations give the generalization of (|3]l (these are also pre- 
cisely the replica equations) 

yi,i+k = ^i,i+fe"^~^ nLio,fe'#fc ^i+k-k' ,i+k~' 

Zi^kd = tanh (ji + Y!k'to,k'^k tanh"^ Ui-k,i-k+k' 

f L , ^l-l -1 



2r+i J2,=-L tanh ( h + J2k=o tanh ^ yi,^+k 



UJ 



It is helpful to keep in mind the message passing interpretation 
of these equations: Ui.i+k are the messages passed from check 
nodes at position i + k into variable nodes at position i and 
Zi-k.i are the messages passed from variable nodes at position 
i — k into checks at position i. Note that the boundary condition 
has now become ^ = 1 for i—k < —L and i+k > L. Note 
also that a'^ j has expressed in terms of other variables and 
therefore eliminated. Once the non-trivial fixed point solution 
is computed for fixed w, replacing it into Gi^com one finds 



Gi,com(w) = hm / 



L+l-l 



^ + I\iJoZi-k/> 



y -In 



— L 



E 

i=-L 



l-l 



l-l 



k=0 



In I e" n (1 + y^+fc) + n (1 - 

L l-l 

E E + yi,t+kZi,i+k) 



i=-L k=0 



— ujh 



This formula constitutes the basis for the numerical results 
described in section HI] Again, we recognize in the first three 
terms the contributions of (2L + 1 + ? — checks, (2L+l)n. 
variable nodes and (2L + edges. 



IV. Analysis of the (/, r, w, L) ensemble 

In this case the combinatorial growth rate G2,com(w) is 
strictly greater than G{uj), and its calculation will not be 
presented here due to lack of space. We use the replica method 
which leads to fixed point equations that are similar to density 
evolution equations, 

^ U? — 1 ^ T — 1 

My) 

'IIT 

fci,--- ,fcr-i=0-' j=l 
r-1 



n,;(z) = 



fci,-,fc.-i=0"' 
r-1 

i=i 

I. u. n A — 1 



X (5(z — tanh(/i + tanh ^ y^) 
with the relation between h and 

i 

UJ = 



„■ — T z. ... 7... — n — 1 



X 



tanh(/i + tanh ^ 



The messages emanating from variable nodes at position i 
and entering checks in the range {i, - ■ ■ ,i + w — 1} have 
density Ili{z). Those emanating from the checks at position 
i and entering variables in the range {i — w + 1, - ■ ■ ,i} have 
density Ili{y). Because of the randomized construction of 
the ensemble these densities are non trivial and the above 
equations do not reduce to equations for scalar variables. For 
the boundary conditions we have to set Ili{z) = S{z — 1) for 
i e {-L-w + l, - ■ ■ ,-L-l}andi e {L + 1,-- - ,L+w-l}. 

Once these equations are solved the most convenient way 
to obtain the growth rate is to use the integral relation (|6). 
Alternatively one can use the expression, 



1 

G2{uj) = -ujh + lim 

i=-l 

ki,---,kr=Q j=l 
^ w — 1 „ I 

kt,-,ki=0 j = i 

/I i 

^ J=l 3=1 

^ w—1 „ ; 

X! / n dyjdzjfli+k{'y)'^i{z) ln(l + yz) 



fc=0 " j=l 



A simplification of this formalism occurs in the limit of 
large I, r with R = 1 — \ fixed. Indeed the densities become 



Dirac distributions and the set of fixed point equations takes 
a form similar to those the density evolution equations of 
coupled LDPC ensemble for the binary erasure channel. A 
similar fact has been discussed in lITSi for constraint satisfac- 
tion problems such as If -SAT and Q coloring for large K and 
Q. Details of this analysis will appear elsewhere. 
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